Jonathan Lebon [Thu, 19 Apr 2018 13:58:51 +0000 (09:58 -0400)]
tests: Port the last python2 script to python3
This is not used by any test, nor is it packaged. Though let's just port
it over to py3 to certify our codebase completely py2-free. I've
manually checked that the script is still functional.
Jonathan Lebon [Thu, 19 Apr 2018 14:16:35 +0000 (10:16 -0400)]
ci: Temporary libgcrypt workaround for older kernels
I'm trying to get ostree tests to pass in OpenShift as part of our CI
move but I've been seeing lots of failures related to GPG tests. It
finally turned out to be because libgcrypt doesn't behave well on older
kernels that don't have `getrandom()` (the cluster is running on RHEL7).
Thankfully, there's a new build with a fix for this. Pull that in
manually until it gets into stable.
For more information, see:
https://bugzilla.redhat.com/show_bug.cgi?id=1542453
Colin Walters [Thu, 19 Apr 2018 19:53:58 +0000 (15:53 -0400)]
tests/installed: Make reboot task less racy
This took a whole lot of experimentation. I hit upon the idea
of doing a `systemctl stop sshd` to avoid the situation where we
might ssh back into the system while it's in the process of shutting
down.
Ultimately the other fix is disabling `ControlMaster`; see
for example: https://github.com/ansible/ansible/issues/17935
Matthew Leeds [Mon, 26 Mar 2018 07:01:39 +0000 (00:01 -0700)]
create-usb: Update summary in destination repo
Currently the create-usb command only generates a summary file in the
destination repo if one doesn't already exist, which means if one does
exist it becomes out of date after the new refs are pulled. This commit
makes ostree regenerate the summary regardless of whether it exists, so
that consumers such as ostree_repo_find_remotes_async() (and at a higher
level, GNOME Software) get an accurate picture of the refs available on
the mount. This commit also updates one of the unit tests to check that
the summary is accurate after a second pull into the same repo.
Since any user of the create-usb command is using collection IDs they
are new enough to be using the unsigned summary support. While it would
technically be possible to use summary signatures on a repo and use the
create-usb command on it (a scenario broken by this commit), the
create-usb command is designed for P2P distribution of refs, which
requires use of unsigned summary support. So this is a legitimate
narrowing of the tool.
Sam Spilsbury [Tue, 16 Jan 2018 01:23:11 +0000 (09:23 +0800)]
avahi: Don't complain with g_warning if the daemon wasn't running
This is a normal case when running unit tests in client code
on continuous integration infrastructure. When those tests are
running they will set G_DEBUG=fatal-warnings which will cause
the program to abort if a warning is emitted. Instead, emit
a debug message if the problem was that we couldn't connect to
the daemon.
Colin Walters [Thu, 12 Apr 2018 16:40:08 +0000 (12:40 -0400)]
lib/sysroot: Move staged into deployment list, rework handling
Followup to: https://github.com/ostreedev/ostree/pull/1503
After starting some more work on on this in rpm-ostree, it is
actually simpler if the staged deployment just shows up in the list.
It's effectively opt-in today; down the line we may make it the default,
but I worry about breaking things that e.g. assume they can mutate
the deployment before rebooting and have `/etc` already merged.
There's not that many things in libostree that iterate over the deployment
list. The biggest change here is around the
`ostree_sysroot_write_deployments_with_options` API. I initially
tried hard to support a use case like "push a rollback" while retaining
the staged deployment, but everything gets very messy because that
function truly is operating on the bootloader list.
For now what I settled on is to just discard the staged deployment;
down the line we can enhance things.
Where we then have some new gymnastics is around implementing
the finalization; we need to go to some effort to pull the staged
deployment out of the list and mark it as unstaged, and then pass
it down to `write_deployments()`.
This is a version of ostree_repo_traverse_commit_union that also
remembers where the objects came from, by recording the parent
relationships in a hashtable. This can be used to later find which
commits each object was from, which we want to use in fsck.
Differently than the original implementation, now we don't attempt any
test for reflinks support on the parent repository, since the test
requires write access to the repository.
Additionally, also check that the two repositories are on the same
device before attempting any reflink.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Closes: #1525
Approved by: cgwalters
Colin Walters [Tue, 3 Apr 2018 20:22:31 +0000 (16:22 -0400)]
switchroot: Don't log if running as pid1, minor code style cleanups
If we're running as pid1, avoid printing anything in the normal
success paths as we don't want to affect the physical console by
default; the device may be using a splash screen, etc.
Also cleanup the code a bit to use a single variable
`running_as_pid1`, declare-and-initialize, use the
`bool` type, etc.
Colin Walters [Thu, 12 Apr 2018 16:53:17 +0000 (12:53 -0400)]
deploy: Silently do nothing if passed same set of deployments
Prep for handling staged deployments better; if we're not passed
the staged one back, then we just want to delete it but not
touch the bootloader config.
Colin Walters [Thu, 12 Apr 2018 16:43:39 +0000 (12:43 -0400)]
deploy: Clean up bootserial assignment function
The reason we were returning a hashtable is a bit lost to history,
there's no reason to do so now anyways. Also port to declare-and-initialize
style and add more comments.
Colin Walters [Thu, 22 Feb 2018 20:27:59 +0000 (15:27 -0500)]
Add concept of "staged" deployment
Add API to write a deployment state to `/run/ostree/staged-deployment`,
along with a systemd service which runs at shutdown time.
This is a big change to the ostree model for hosts,
but it closes a longstanding set of bugs; many, many people have
hit the "losing changes in /etc" problem. It also avoids
the other problem of racing with programs that modify `/etc`
such as LVM backups:
https://bugzilla.redhat.com/show_bug.cgi?id=1365297
We need this in particular to go to a full-on model for
automatically updated host systems where (like a dual-partition model)
everything is fully prepared and the reboot can be taken
asynchronously.
Jonathan Lebon [Wed, 11 Apr 2018 20:45:40 +0000 (16:45 -0400)]
ci: turn gating back on for most testsuites
Somehow, this slipped through in #1513. We weren't inheriting anymore,
so `branches` defaulted back to just `master`. This means we weren't
gating on most of the containerized builds anymore. Ouch!
It'd make sense to teach PAPR to allow some defaults across all
testsuites, even in the `inherit: false` case. Though it's tempting to
also just change the hardcoded PAPR default to those branches since our
use of Homu + PAPR at this point is pretty ubiquitous, and it doesn't
really hurt for the ones that don't use it.
Owen W. Taylor [Wed, 11 Apr 2018 22:06:39 +0000 (18:06 -0400)]
Don't scan uncompressed_objects_dir if it doesn't exist
A newly created archive-mode repository won't have a uncompressed-objects-cache
directory, and uncompressed_objects_dir is -1 to flag that. The special meaning of
-1 meaning "cwd" for libglnx means that the current directory was scanned as
if it was an objects directory, producing unexpected results, especially if there
were any two-letter files/subdirs in the current directory.
Matthew Leeds [Wed, 28 Mar 2018 07:31:05 +0000 (00:31 -0700)]
lib/repo: Add timestamps to OstreeRepoFinderResult
Currently OstreeRepoFinderResult, a data structure used by pull code
that supports P2P operations, has a hash table mapping refs to checksums
but doesn't include timestamp information. This means that clients have
no way of knowing just from the OstreeRepoFinderResult information if a
commit being offered by a peer remote is an update or downgrade until
they start pulling it. The client could check the summary or the commit
metadata for the timestamps, but this requires adding the temporary
remotes to the repo config, and ostree is already checking timestamps
before returning the results, so I think it makes more sense for them to
be returned rather than leaving it to the client. This limitation is
especially important for offline computers, because for online computers
the latest commit available from any remote is the latest commit,
period.
This commit adds a "ref_to_timestamp" hash table to
OstreeRepoFinderResult that is symmetric to "ref_to_checksum" in that it
shares the same keys. This is an API break, but it's part of the
experimental API, and none of the current users of that (flatpak,
eos-updater, and gnome-software) are affected. See the documentation for
more details on "ref_to_timestamp". One thing to note is the data
structure currently gets initialized in find_remotes_cb(), so only users
of ostree_repo_find_remotes_async() will get them, not users of, say,
ostree_repo_finder_resolve_all_async(). This is because the individual
OstreeRepoFinder implementations don't currently access the timestamps
(but I think this could be changed in the future if there's a need).
This commit will allow P2P support to be added to
flatpak_installation_list_installed_refs_for_update, which will allow
GNOME Software to update apps from USB drives while offline (it's
already possible online).
pull: Don't save summary to cache before validating signatures
In case of some kind of race or other weirdness we might be getting
non-matching versions of summary.sig and summary, where summary.sig
is the latest version. Currently we're saving them to the cache
directly after downloading them successfully, but they will then fail
to gpg validate. Then on the next run we'll keep using the cached files
even if they are incorrect, until summary.sig changes upstream.
This changes the order so that we verify the signatures before saving
to the cache, thus ensuring that we don't end up in a stuck state.
Matthew Leeds [Thu, 29 Mar 2018 07:03:43 +0000 (00:03 -0700)]
lib/repo-pull: Improve error message when no summary is found
In ostree_repo_remote_fetch_summary_with_options(), if no summary is
found on the server and summary verification is enabled, the error
message implies that it's the summary signature that's missing, which is
misleading. This commit adds a more specific error message for the case
of a missing summary, which has the side effect of explicitly checking
for the case that signatures != NULL && summary == NULL after
repo_remote_fetch_summary(), even though that should never happen.
One effect of this is that if you run "flatpak remote-add" with an
incorrect URL you get a more helpful error message, and similarly for
other flatpak operations and other users of ostree.
Matthew Leeds [Sat, 31 Mar 2018 01:11:45 +0000 (18:11 -0700)]
lib/repo-pull: Rename a variable for clarity
In libostree, the phrase "commit metadata" has two meanings-- one is the
first dictionary in a commit GVariant that stores metadata such as ref
bindings, and the other is the commit metadata in the summary file,
which stores the commit size, checksum, and timestamp. In
find_remotes_process_refs(), the entire commit GVariant was being
referred to as commit metadata, so this commit changes the variable
name and a comment to make things more consistent.
In _try_clone_from_payload_link, don't try to do the clone in the
parent repo, because we don't want to modify that. parent repos are
typically used when you want a shared, immutable base.
For example in flatpak, the parent repo is the system repo which you
don't have write access to, so any modification to it will fail with
EACCES, making it impossible to install via the system helper.
Matthew Leeds [Thu, 29 Mar 2018 06:11:50 +0000 (23:11 -0700)]
libotutil/checksum-utils: Fix memory management
Ostree uses the OtChecksum data structure as a wrapper around GChecksum
(depending on what libraries are available at compile time). According
to the docs for g_checksum_get_digest(), a GChecksum value can no longer
be updated after that function is called. Ostree enforces this by
setting "initialized" to FALSE after getting the digest, but this leads
to ot_checksum_clear() avoiding freeing any memory, leading to leaks. So
this commit adds a "closed" value that gets set when getting a digest
and checked when updating the value, so the initialized value can be
used only for memory management.
Colin Walters [Mon, 26 Mar 2018 17:23:25 +0000 (13:23 -0400)]
tests/installed: Fix TESTS= being empty
I broke this in https://github.com/ostreedev/ostree/pull/1509/commits/9b55aaea6f34b7094c44932a3c2e1cf2d54634fd
I'd obviously tested *setting* it locally worked, but I didn't test that
not having it set ran all the tests.
I don't understand why we were doing the `+ ` pattern before; let's
just check if it's empty.
Colin Walters [Fri, 23 Mar 2018 20:02:38 +0000 (16:02 -0400)]
lib/deploy: Port final bootconfig writing to new style
The main blocker for doing this before was the `goto out` handling
for remounting `/boot`. Handle that by factoring out the bits that
require it to a helper function, and do the C/GError equivalent of
"try/finally".
Not prep for anything right now, just decided to do this since I had the file
open.
Colin Walters [Fri, 23 Mar 2018 19:33:38 +0000 (15:33 -0400)]
lib/deploy: Split /etc merge into two stages
For staged deploy, we want to pay the cost of creating copies from
`/usr/etc` → `/etc` at stage time, since it can be expensive. (We
want to minimize time spent during shutdown).
Split it up into two functions; the logic is also simply clearer.
Colin Walters [Fri, 23 Mar 2018 19:25:14 +0000 (15:25 -0400)]
lib/deploy: Set kargs in one place
Prep for deployment staging. We had the code to hande "explicit kargs" in one
place, but the "use merge deployment" karg bits mixed in with the "/etc merge"
logic. Those are separate things, and it's better to have karg handling in one
place.
Colin Walters [Thu, 22 Mar 2018 20:52:04 +0000 (16:52 -0400)]
Release 2018.4
A quick turnaround to include one PR: https://github.com/ostreedev/ostree/pull/1508
"switchroot: Ensure /run/ostree-booted is created even without initramfs"
This fixes ostree when booting without an initramfs. Thanks to @akiernan for the
bug report and helping review the fix! I'm working on enhancing
the test suite, which will help in adding some coverage here.
Also for this release I'm going to avoid adding a "stub" symbol section
to the `-released.sym` file; I don't believe it's necessary.
Colin Walters [Thu, 22 Mar 2018 14:14:22 +0000 (10:14 -0400)]
switchroot: Ensure /run/ostree-booted is created even without initramfs
See https://mail.gnome.org/archives/ostree-list/2018-March/msg00012.html
If ostree-prepare-root is run as pid 1 (i.e we're not using an initramfs), then
anything we write outside the target sysroot (such as `/run/ostree-booted`) will
be lost.
Since `ostree-remount.service` runs fairly early in boot, and is triggered via
`ConditionKernelCommandLine=ostree`, we can just touch the file there in
addition.
Colin Walters [Thu, 22 Mar 2018 15:36:33 +0000 (11:36 -0400)]
tests/installed: Use temporary directories more consistently
This is prep for splitting off "nondestructive" tests which
we can run in parallel from the destructive/invasive ones which
e.g. change the host refspec, do deployments.
The `cd` invocation in `prepare_tmpdir` wasn't working because we were running
it in a subshell. Fix this by dropping the subshell.
Colin Walters [Wed, 21 Mar 2018 14:14:12 +0000 (10:14 -0400)]
Release 2018.3
It's been over a month since 2018.2; we have a few features and various fixes,
and the "stage" work pending which is pretty invasive. Time for a new release!
Jonathan Lebon [Wed, 14 Mar 2018 14:36:48 +0000 (10:36 -0400)]
lib/fetcher: Allow clients to append to User-Agent
We do already have `http-headers`, which potentially could be used to
allow clients to completely override the field, but it seems like the
more common use case is simply to append.
Jonathan Lebon [Fri, 16 Mar 2018 18:19:28 +0000 (14:19 -0400)]
lib/core: Support <remote>: syntax when listing refs
Allow users to pass `<remote>:` to list all refs we have locally
belonging to `<remote>`. Also (re-)allow the similar `<remote>:.` syntax
for backwards compatibility with flatpak.
Colin Walters [Fri, 9 Mar 2018 18:26:07 +0000 (13:26 -0500)]
tests/str: Rework invocation
Let's make our `run.sh` generically support any playbook. This is prep for
writing further tests in Ansible. Along with that, rework the Ansible so that
`tests.yml` is a playbook, and then the other bits are just task lists. It's
easier to read.
I also started to add a `use_git_build` variable with the idea that we'll be
able to run these same tests against an upstream image by setting that variable
off.
Colin Walters [Mon, 12 Mar 2018 18:55:51 +0000 (13:55 -0500)]
sysroot: Rework how we find booted deployment
I was looking at this code in prep for "staging" deployments,
and there are several cleanups to be made here. The first
thing I noticed is that we look for the `ostree=` kernel argument,
but the presence of that should be exactly equivalent to having
`/run/ostree-booted` exist. We just added a member variable for
that, so let's make use of it.
Related to this, we were erroring out if we had the karg but
didn't find a deployment. But this can happen if e.g. one is
using `ostree admin --sysroot` from an ostree-booted system! It's
actually a bit surprising no one has reported this so far; I guess
in the end people are either using non-ostree systems or running
from containers.
Let's add a member variable `root_is_sysroot` that we can use
to determine if we're looking at `/`. Then, our more precise
"should find a booted deployment" state is when both `ostree_booted`
and `root_is_sysroot` are TRUE.
Next, rather than walking all of the deployments after parsing,
we can inline the `fstatat()` while parsing. The mild ugly
thing about this is assigning to the sysroot member variable while
parsing, but I will likely clean that up later, just wanted to avoid
rewriting everything in one go.
Rasmus Thomsen [Thu, 8 Mar 2018 16:20:11 +0000 (17:20 +0100)]
configure: add option for libsystemd
Until now ostree checked for libsystemd and enabled
support for it if it found it. This commit changes that
behavior by adding an option to enable/disable libsystemd.
This is especially useful if one uses a source based distro
(like Gentoo/Exherbo), where one wants to avoid such automagic
detection of dependencies and prefers switches for that instead.
Colin Walters [Mon, 12 Mar 2018 22:02:04 +0000 (18:02 -0400)]
tests: Avoid generating lots of output in itest-payload-link
We noticed this in a recent PR. While I'm here, also only do
the `find` once, add `-type l` for good measure, and use our
built in `libtest.sh` assertion functions.
Joaquim Rocha [Mon, 12 Mar 2018 12:24:22 +0000 (13:24 +0100)]
pull: Ignore the cancellable when aborting a transaction
In ostree_repo_abort_transaction, if we pass a cancellable and it gets
canceled, then the function may fail to fully clean up the transaction
state. This was happening e.g. when the ostree_repo_pull_with_options
call got cancelled.
To fix this, as suggested by Colin Walters, we set the passed
cancellable as NULL, in order for it to be ignored.
Colin Walters [Thu, 22 Feb 2018 19:16:33 +0000 (14:16 -0500)]
ci: Rework installed tests to use Fedora Standard Test interface
Reusing the way `standard-test-roles` has support for booting
a qcow2 actually gets us to the "VM-in-container" flow. Plus
Ansible over shell script is sometimes nicer.
It's better than what we were doing before for installed tests,
and moreover using Ansible more broadly for testing is going
to align us better with Fedora's CI.
As part of this I split off a "libpaprci" which I intend to maintain
as a "copylib" for a little bit between ostree/rpm-ostree, and then
we'll figure out how to expand from there (maybe some of the patterns
get "baked in" to PAPR for example).
Note the `FAH27-insttests` context moves to the top since it's now
of primary importance, and I expect that we start expanding it.
Jeremy Hiatt [Thu, 8 Mar 2018 01:44:43 +0000 (01:44 +0000)]
lib/repo: Fix multi-signature support when generating summary files
Ensure that the metadata object is built up with the signatures from all keys
passed to ostree_repo_add_gpg_signature_summary(). Previously only the signature
from the last key would end up in the metadata.
When a new object is added to the repository, create a
$PAYLOAD-SHA256.payload-link symlink file as well. The target of the
symlink is the checksum of the object that was added the repository.
Whenever we add a new object file, in addition to lookup if the file is
already present with the same checksum we also check if an object with
the same payload is in the repository.
If a file with the same payload is already present in the repository, we
copy it with `glnx_regfile_copy_bytes` that internally attempts to
create a reflink (ioctl (..., FICLONE, ..)) to the target file if the
file system supports it. This enables to have objects that share the
payload but have a different inode and xattrs.
By default the payload-link-threshold value is G_MAXUINT64 that disables
the feature.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Closes: #1443
Approved by: cgwalters
Colin Walters [Mon, 5 Mar 2018 22:31:12 +0000 (17:31 -0500)]
repo/refs: Clean up error prefixing
Add some "function global" prefixing in line with what we do in
other places now, and drop the "manual filename" prefixing that
is no longer necessary since
https://github.com/GNOME/libglnx/commit/23f7df15006f14ddc3bc2ddee690f7f8604c3ebe
Matthew Leeds [Sat, 3 Mar 2018 20:57:12 +0000 (12:57 -0800)]
lib: Fix memory leaks of OstreeRemote
The _ostree_repo_get_remote() and _ostree_repo_get_remote_inherited()
methods transfer ownership of the returned OstreeRemote to the caller,
so this commit fixes a few call sites that weren't properly freeing it.
Colin Walters [Mon, 5 Mar 2018 15:42:19 +0000 (10:42 -0500)]
prune: Error if --static-deltas-only without --delete-commit
The original changes here apparently had the *idea* that `--static-deltas-only`
would be useful in general, but we never implemented that. The current
situation where it's ignored unless `--delete-commit` is specified is
very misleading and I can easily see it leading to data loss for people.
Let's error out until we have a chance to make it actually useful.
Colin Walters [Mon, 5 Mar 2018 15:56:45 +0000 (10:56 -0500)]
lib/repo: Do account for size with prune --no-prune
I think this got changed in a refactor. We definitely want
to total up the amount of space that *would* be freed even
with `--no-prune` AKA `OSTREE_REPO_PRUNE_FLAGS_NO_PRUNE`.
It's actually a bit terrifying this is apparently the first test case for
the `--no-prune` option...
Matthew Leeds [Thu, 1 Mar 2018 00:06:31 +0000 (16:06 -0800)]
lib/repo-pull: Fix free function for hash table
The "ref_original_commits" hash table uses string values, not variants,
so fix the free function passed to g_hash_table_new_full (). Since
g_variant_unref isn't NULL safe, this prevents an assertion failure when
a NULL value is inserted.
Dan Nicholson suggested this patch; I'm just submitting it because he's
busy.
If we're booted into a deployment, then any queries for the pending
merge deployment of a non-booted OS will fail due all of them being
considered rollback.
Fix this by filtering by `osname` *before* determining if we've crossed
the booted deployment yet.
Matthew Leeds [Tue, 27 Feb 2018 02:54:54 +0000 (18:54 -0800)]
lib/repo-finder-mount: Fix path to flatpak repo
OstreeRepoFinderMount checks mounts for a few well-known directories
such as "ostree/repo" and ".ostree/repo" to try to find remotes. One of
the hard-coded directories is "var/lib/flatpak" but that's the flatpak
directory, not the ostree repo used by flatpak, which is at
"var/lib/flatpak/repo". So this commit changes the path so the repo can
be found.
For recent versions of Endless, flatpak uses /ostree/repo as its
repository, so this commit won't make a difference there. But it may
help on other operating systems.
Colin Walters [Mon, 26 Feb 2018 19:11:00 +0000 (14:11 -0500)]
bash-completion: Remove `admin` completions
The `admin` commandline should be considered a demo; I just added
the `pin` command *mostly* so we could use it for unit tests, although
I can imagine other people using it.
But maintaining completions is a lot of overhead right now, let's not
do it for `admin`.
The other command line options that operate on repos we will definitely maintain
since they're used in releng contexts.
Colin Walters [Fri, 23 Feb 2018 17:46:32 +0000 (12:46 -0500)]
sysroot: Add concept of deployment "pinning" 📌
Example user story: Jane rebases her OS to a new major version N, and wants to
keep around N-1 even after a few upgrades for a while so she can easily roll
back. I plan to add `rpm-ostree rebase --pin` to opt-in to this for example.
Builds on the new `libostree-transient` group to store pinning state there.
Colin Walters [Fri, 23 Feb 2018 19:23:38 +0000 (14:23 -0500)]
sysroot: Add API to clean up transient keys in origin files
The `origin/unlocked` and `origin/override-commit` keys are examples of state
that's really transient; we don't want to maintain them across upgrades. Right
now there are bits for this in both `ostree admin upgrade` as well as in
rpm-ostree.
This new API will slightly clean up both cases, but it's really prep for adding
a concept of deployment "pinning" that will live in the new
`libostree-transient` group.
projects / ostree.git / log